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Abstract 


This  paper  will  evaluate  cost  overruns  from  a  microeconomic  perspective  to 
determine  their  root  causes.  The  specific  variables  that  will  be  evaluated  are:  contract 
budget  fluctuations,  contract  length,  inflation,  procurement  budget  fluctuation,  research 
and  development  budget  fluctuation,  the  technology  readiness  of  the  commodity,  and 
industry  concentration.  These  variables  will  be  evaluated  twice.  The  first  evaluation  will 
consist  of  a  binary  choice  model  to  determine  whether  or  not  the  dependent  variables 
influence  the  likelihood  of  a  cost  overrun.  The  specific  form  of  the  evaluation  will  take 
the  form  of  a  probit  regression  in  which  an  independent  variable  value  of  zero  indicates 
the  contract  did  not  overrun  its  initial  contract  budget  baseline  and  one  indicates  the 
contract  cost  more  than  its  initial  baseline.  The  second  evaluation  will  regress  the 
dependent  variables  against  the  natural  logarithm  of  the  magnitude  of  a  cost  overrun  to 
determine  whether  or  not  they  influence  the  amount  a  contract  overruns  its  initial 
baseline.  This  study  will  show  that  budget  variability,  inflation,  industry  consolidation 
and  immature  technologies  increase  the  likelihood  that  a  contract  will  overrun  its  budget. 
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COST  GROWTH  IN  WEAPONS  SYSTEMS:  RE-EXAMINING  THE 
RUBBER  BASELINE  AND  ECONOMIC  FACTORS 


I.  Introduction 


Issue 

Defense  contracts  have  been  a  part  of  the  military  establishment  since  the 
Revolutionary  war.  Defense  contract  cost  overruns  have  most  likely  existed  as  long  as 
defense  contracts.  However,  prior  to  World  War  II,  government  agencies  were  not  held 
accountable  for  exceeding  program  estimates  (Calcutt,  1993).  In  contrast  to  this  previous 
policy,  Congress  has  required  the  Department  of  Defense  (DOD)  to  explain  virtually  all 
cost  overruns  since  the  end  of  World  War  II. 

The  nature  of  national  defense  makes  defense  contract  cost  overruns  a  concern  for 
all  tax-paying  adults.  National  defense  is  a  public  good.  As  such,  citizens  can  neither 
opt  out  of  its  protection  nor  is  there  a  viable  alternative  means  of  protection.  Therefore, 
the  financial  costs  of  a  defense  contract  cost  overrun  are  shared  by  all. 

Cost  overruns  are  the  result  of  two  possible  root  causes.  The  first  is  simply  the 
cost  of  producing  the  weapon  system  was  more  than  anticipated.  The  second  possible 
explanation  for  cost  overruns  is  that  recently  developed  technology  is  incorporated  into 
existing  contracts  thereby  expanding  the  contract’s  scope. 
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Hypothesis 


Traditionally,  a  cost  variance  (CV)  has  been  defined  as  the  difference  between  the 
budgeted  cost  of  work  perfonned  (BCWP)  and  the  actual  cost  of  work  performed 
(ACWP)  (Gordon,  1996).  This  CV  is  then  compared  to  the  current  contract  budget 
baseline  to  determine  cost  and  schedule  perfonnance.  According  to  this  construct,  a  cost 
overrun  occurs  when  the  value  of  the  CV  is  negative. 

However,  this  is  a  technical  definition  utilized  by  the  DOD  that  is  overlooked  by 
Congress  and  the  American  taxpayer.  A  more  appropriate  definition  of  a  cost  overrun 
should  compare  the  current  estimate  at  completion  (EAC)  to  the  initial  contract  budget 
base  (CBBI).  This  result  more  accurately  captures  the  cost  growth  of  the  program  from 
inception  to  finish,  rather  than  from  an  arbitrary  point  along  the  contract’s  evolution. 

This  study  proposes  a  two-step  method  to  detennine  the  impact  and  magnitude  of 
DOD  contract  cost  overruns.  This  process  is  based  on  the  method  used  by  Vincent 
Sipple  in  his  2002  thesis  on  engineering  cost  risk.  The  first  step  is  to  determine  the  right- 
hand  side  (RHS)  variables  that  influence  whether  or  not  a  contract  will  overrun  its  initial 
budget.  The  second  step  is  to  detennine  the  RHS  variables  that  influence  the  magnitude 
of  a  realized  cost  overrun.  Both  steps  postulate  that  the  likelihood  and  magnitude  of  cost 
overruns  are  a  function  of  inflation,  the  number  of  times  a  contract’s  budget  changes, 
changes  in  the  DOD  Procurement  and  Research,  Development,  Test  and  Evaluation 
(RDT&E)  budgets,  the  concentration  of  the  specific  segment  of  the  defense  industry,  the 
technological  maturity  of  the  commodity  being  purchased,  and  the  length  of  the  contract. 
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Scope  and  Limitations  of  the  Study 


The  data  used  in  this  study  was  derived  from  the  Defense  Acquisition  Executive 
Summary  database  extract  used  by  James  Smirnoff  in  his  2006  study  of  the  defense 
industry.  The  database  contained  contracts  spanning  1970  to  2002.  It  included  14,004 
progress  reports  from  the  individual  services  covering  1,150  contracts  (Smirnoff,  2006). 
The  data  does  not  provide  insight  as  to  why  a  contract  budget  baseline  changed. 

Therefore  the  results  generated  by  this  study  can  not  provide  any  insight  into  which  of 
the  two  causes  of  cost  overruns  is  more  prevalent. 

Research  Objectives 

As  stated  above,  the  goal  of  this  study  is  twofold.  The  first  objective  is  to  identify 
what  causes  a  defense  contract  to  overrun  its  initial  budget.  The  second  objective  of  this 
study  is  to  determine  what  variables  influence  the  magnitude  of  a  contract  cost  overrun. 
This  study  utilized  a  probit  regression  to  evaluate  the  causes  of  a  cost  overrun  while  a 
ordinary  least  squares  (OLS)  regression  was  used  to  determine  which  variables  influence 
the  magnitude  of  a  cost  overrun. 

Thesis  Overview 

The  remainder  of  this  thesis  will  be  organized  according  to  the  traditional  thesis 
five  chapter  format.  Chapter  II  will  explore  the  existing  literature  to  explain  why  the 
RHS  variables  were  chosen  to  evaluate  the  causes  and  magnitudes  of  defense  contract 
cost  overruns.  Chapter  III  builds  the  model  and  methods  used  to  evaluate  the  cost 
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overruns  and  magnitudes.  Chapter  IV  reports  the  models’  results.  Finally,  Chapter  V 
presents  a  conclusion  and  recommends  follow-on  research. 
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II.  Literature  Review 


Basis  for  Study 

The  public  good  properties  of  defense  contracts  have  given  rise  to  a  great  amount 
of  research  into  the  nature  of  defense  contract  cost  overruns.  However,  the  bulk  of  this 
research  has  isolated  specific  variables  in  the  earned  value  management  system  rather 
than  using  economic  principles  to  focus  their  analysis.  Despite  this  lack  of  a  theoretical 
underpinning,  a  common  theme  that  runs  through  defense  acquisition  literature  is  the 
notion  that  program  instability  leads  to  contract  cost  overruns.  Program  instability  can  be 
caused  by  changing  requirements,  micro-  and  macro-budget  instability,  and  immature 
technology. 

Recent  cost  analysis  research  has  explored  virtually  every  aspect  of  the  earned 
value  management  system  and  acquisition  reform.  Therefore,  this  chapter  will  focus  on 
the  effect  of  program  instability  on  individual  contracts  and  economic  theory  with 
specific  emphasis  on  macroeconomic  variables.  The  goal  is  to  develop  the  theoretical 
basis  for  the  model  of  cost  overruns  proposed  in  chapter  I.  The  specific  RHS  variables 
that  will  be  discussed  are:  inflation,  the  number  of  times  a  contract  is  rebaselined,  DOD 
budget  instability,  industry  concentration,  contract  length,  and  technology  readiness. 

Inflation 

The  first  independent  variable  to  be  explored  is  national  inflation.  Basic 
economic  theory  holds  that  inflation  and  industry  concentration  both  affect  product  price 
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levels.  According  to  the  DOD  in  a,  . .  macroeconomic  sense  inflation  is  the  change  in 
buying  power  over  time”  (MCR,  2006).  Smirnoff,  in  his  2006  study,  evaluated  the 
effects  of  unexpected  inflation  on  all  DOD  contracts  in  a  given  year.  He  defined 
unexpected  inflation  as  the  difference  between  forecast  inflation  and  what  actually 
occurred.  However,  his  analysis  determined  that  unexpected  inflation  did  not  influence 
cost  overruns  (Smirnoff,  2006). 

Rebaselines 

The  second  RHS  variable  evaluated  in  this  analysis  is  the  instability  of  individual 
contract  budgets.  Contract  budget  instability  is  epitomized  by  the  number  of  times  a 
contract  has  been  rebaselined.  James  Gordon,  in  a  1996  thesis,  produced  what  is 
probably  the  best  example  of  academic  research  into  the  effect  of  contract  instability  on 
final  price.  In  this  study,  he  defined  a  rebaseline  as,  “increases  or  decreases  in  the 
contract  budget  baseline  (CBB)  greater  than  10%  or  adding  up  to  greater  than  10%  of  the 
CBB  prior  to  the  change”  (Gordon,  1996).  Unfortunately,  this  definition  can  mask  a 
significant  amount  of  program  instability  if  a  program  encounters  a  multitude  of  small 
changes  that  do  not  aggregate  to  a  large  percentage  of  the  contract,  or  they  aggregate  to 
only  one  or  two  10  percent  changes.  Despite  the  rigor  of  his  research,  Gordon  was 
unable  to  conclude  that  rebaselines  influence  the  likelihood  of  a  cost  overrun  (Gordon, 
1996).  These  results  contradicted  the  existing  literature  at  the  time  of  his  study. 
However,  a  more  precise  definition  of  a  rebaseline  may  have  yielded  different  results. 

A  second  more  recent  study  into  the  effect  of  rebaselines  on  contract  cost  and 
schedule  was  conducted  by  Steven  Cross  in  2006.  This  study  was  primarily  focused  on 
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validating  the  quality  of  the  data  contained  in  the  selected  acquisition  reports  on  major 
defense  acquisition  programs  (MDAP).  A  subsidiary  goal  of  this  study  was  to  add  new 
variables  to  recent  cost  research.  Two  of  these  new  variables  were  the  number  of  times  a 
MDAP  was  rebaselined  and  whether  an  MDAP  was  rebaselined  two  or  more  times. 

Cross  concluded  that  the  number  of  rebaselines  for  MDAP  does  predict  schedule  growth 
and  two  or  more  rebaselines  predicts  cost  growth  (Cross,  2006).  However,  this  study 
analyzed  MDAPs  as  a  whole  and  not  as  individual  contracts.  Each  MDAP  is  comprised 
of  multiple  contracts;  therefore  program  managers  can  move  money  between  the 
contracts  without  changing  the  overall  program  budget.  As  such,  analyzing  changes  in 
the  MDAP  budget  may  hide  the  effects  on  individual  contracts. 

Finally,  the  Government  Accountability  Office  (GAO)  and  the  U.S.  Congress 
have  also  been  interested  in  the  effect  of  program  rebaselines  on  the  final  cost  of 
MDAPs.  In  2005,  a  GAO  audit  determined  that  rebaselining  MDAPs  can  mask  the  real 
per  unit  cost  of  new  weapon  systems.  This  phenomenon  occurs  because  the  number  of 
units  purchased  is  decreased  to  prevent  the  need  to  increase  the  contract  budget.  Further, 
once  a  program  is  rebaselined  all  earned  value  management  information  is  reset  and  price 
and  schedule  increases  are  erased.  Therefore,  cost  overruns  based  on  the  original 
estimate  are  no  longer  reported  (Levin,  2005). 

DOD  Budgets 

Since  defense  is  a  public  good,  there  is  only  a  fixed  amount  of  national  treasure  to 
finance  the  DOD.  Therefore,  the  funding  of  all  DOD  contracts  and  operations  is  a  classic 
budget  constraint  problem.  Axel  Gautier  studied  a  procurement  environment  where  the 
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provider  is  paid  before  the  project  commences.  This  situation  exemplifies  DOD 
acquisition  programs.  He  stated  that  a  significant  consequence  of  this  practice  is  that  the 
procurer  will  obtain  less  of  the  good  than  desired  (Gautier,  2004).  Therefore,  in  order  to 
obtain  what  is  required  in  the  initial  contract,  the  procuring  agency  must  pay  more  than 
originally  negotiated. 

Smirnoff  also  explored  the  effects  of  fluctuations  in  the  total  DOD  procurement 
and  RDT&E  budgets.  He  determined  that  as  the  DOD  procurement  budget  increases,  the 
volume  of  cost  overruns  decreases,  while  changes  in  the  RDT&E  budget  have  no  effect 
on  the  likelihood  of  cost  overruns  (Smirnoff,  2006). 

Industry  Concentration 

A  cornerstone  of  economic  thought  is  that  competition  lowers  price.  According 
to  Michael  Baye,  monopolies  tend  to,  “restrict  output  and  charge  a  price  above  marginal 
cost”  (Baye,  2003).  According  to  this  proposition,  the  optimal  pricing  situation  exists  in 
a  loosely  concentrated  market.  Therefore,  as  the  defense  industry  becomes  more 
concentrated,  both  negotiated  contract  price  and  final  contract  price  should  increase, 
thereby  increasing  the  likelihood  for  a  contract  to  exceed  its  CBBI. 

There  are  two  main  measures  of  industry  consolidation  that  are  used  in  the 
economic  literature,  the  four-firm  concentration  ratio,  otherwise  known  as  the  CR4,  and 
the  Herfindahl-Hirschman  Index  (HHI)  (Baye,  2003).  The  CR4  is  merely  the  percentage 
of  industry  sales  comprised  by  the  four  largest  producers  in  the  industry.  The  CR8, 
which  is  reported  by  the  U.S.  Census  Bureau,  is  an  extension  of  the  CR4.  It  is  the 


percentage  of  industry  sales  of  the  eight  largest  producers.  The  HHI  squares  the  market 
shares  of  all  firms  in  an  industry  and  then  multiplies  this  value  by  10,000  (Baye,  2003). 

Judy  Davis  conducted  an  analysis  of  the  defense  industry  consolidation.  One  of 
the  areas  of  her  research  was  the  impact  of  consolidation  on  cost  overruns.  She  stated 
that  the  DOD  hoped  to  achieve  more  than  $2B  in  contract  cost  savings  as  the  defense 
industry  consolidated  in  the  1980s  and  1990s  (Davis,  2006).  She  found  that  despite  this 
ambitious  goal,  the  DOD  cannot  verify  that  these  cost  savings  were  achieved.  On  the 
other  hand,  Davis  concluded  that  the  defense  industry  has  become  more  concentrated 
than  in  the  past.  Similarly,  she  detennined  that  profits  to  the  remaining  defense 
contractors  have  increased.  Specifically,  she  calculated  profit  margins,  returns  on  assets, 
and  returns  on  equity  have  all  increased  by  2.0  percent,  1.5  percent,  and  5.9  percent 
respectively  (Davis,  2006).  Unfortunately  she  was  unable  to  tie  these  improvements  in 
financial  fundamentals  to  the  change  in  industry  concentration. 

The  increased  industry  concentration  found  by  Davis  leads  to  less  competition  in 
the  defense  marketplace.  Juan-Jose  Ganuza  explored  the  effect  of  decreased  competition 
in  the  Spanish  defense  market.  He  explored  the  effect  of  infonnation  asymmetry  and 
how  they  relate  to  contract  price  and  the  likelihood  of  a  contract  cost  overrun.  His 
procurement  model  for  markets  with  less  than  perfect  competition  encouraged  the  buyer 
to  provide  very  little  product  specification  and  to  invest  less  than  warranted  in  the 
product  at  the  time  of  contract  award  (Ganuza,  2003).  This  strategy  requires  ongoing 
negotiations  to  refine  the  product  specification  throughout  the  acquisition  process.  This 
strategy  virtually  guarantees  contract  cost  overruns  in  the  highly  technical  defense  market 
(Ganuza,  2003).  He  concluded  that  information  asymmetry  is  a  significant  component  of 
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cost  overruns  and  perfect  competition  leads  to  disclosure  of  all  pertinent  contract 
information  (Ganuza,  2003). 

Technology  Readiness 

Technology  readiness  (TR)  refers  to  the  measurement  of,  “maturity  of  a  particular 
technology  and  the  consistent  comparison  of  maturity  between  different  types  of 
technology”  (Mankins,  1995).  The  Aerospace  Systems  Design  Laboratory  used  TR 
levels  (TRL)  to  classify  emerging  technologies  and  detennine  if  an  aircraft  development 
project  is  economically  viable  as  a  function  of  cost  per  revenue  mile  (Mavris,  1995). 

This  goal  is  accomplished  by  conducting  an  initial  feasibility  study  which  answers  the 
basic  questions  of: 

•  Why  is  the  technology  needed? 

•  Can  off  the  shelf  technology  be  used? 

•  What  is  the  risk  associated  with  the  technology  of  the  project?  (Mavris, 
1995) 

A.  Lee  Battershell  used  this  notion  of  technology  risk  to  compare  Boeing  and  the 
DOD’s  views  on  risk  for  the  development  of  the  C-17.  Boeing  typically  approached 
technology  skeptically,  requiring  new  technology  to  “earn  its  way”  on  a  new  aircraft, 
while  the  DOD  appeared  to  be  preoccupied  with  new  technology  regardless  of  the  cost 
(Battershell,  1995).  Maria  Hedvall  stated  that  engineering  changes  are  relatively 
inexpensive  early  in  a  system’s  production  but  become  increasingly  more  expensive  as 
production  continues  (Hedvall,  2004). 
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Contract  Length 


TR  and  contract  length  are  linked  in  the  literature.  According  to  Battershell,  the 
DOD  approach  to  acquiring  immature  technology  unnecessarily  lengthens  the  acquisition 
process.  New  DOD  acquisition  programs  now  take  1 1  to  21  years  to  complete  because  of 
the  institutional  preoccupation  with  new  technology  (Battershell,  1995).  This  finding  is 
corroborated  by  Hedvall  in  her  study  of  the  Swedish  defense  industry.  She  argued  that 
defense  systems  are  composed  of  several  components  that  must  be  synchronized  to 
produce  the  final  product.  However,  each  component  is  fabricated  at  a  different  rate; 
therefore,  old  technology  must  be  incorporated  into  new  technology.  The  process  of 
incorporating  different  technologies  raised  the  price  of  the  final,  integrated  system 
(Hedvall,  2004). 

Since  the  DOD  acquisition  model  requires  the  government  and  defense 
contractors  to  enter  into  acquisition  programs  without  full  knowledge  of  the  end  product, 
ongoing  contract  negotiations  are  necessary.  Jean  Tirol  advocates  a  two-stage 
procurement  model  where  contractor’s  “buy  into”  a  project.  In  the  first  stage,  the 
contractor  subsidizes  additional  costs  and  enters  into  negotiations  with  the  government  to 
recoup  the  additional  costs  in  the  second  stage  (Tirol,  1986).  Hedvall  agrees  with  the 
need  for  ongoing  contract  negotiation,  although  for  a  different  reason.  She  proposes 
what  she  calls  the  “Culture  of  Change.”  She  asserted  that  this  culture  incorporates  new 
technology  into  ongoing  programs  as  it  becomes  available.  (Hedvall,  2004). 
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III.  Methodology 


Introduction  of  Model 

This  study  utilized  a  two-stage  model  to  evaluate  cost  growth  in  DOD  weapon 
systems.  The  notion  of  using  a  two-step  regression  model  to  identify  the  drivers  of  cost 
and  schedule  growth  and  then  quantify  their  impact  on  the  magnitude  of  a  cost  or 
schedule  overrun  is  not  novel.  A  series  of  theses  at  the  Air  Force  Institute  of  Technology 
beginning  with  Sipple  and  culminating  with  Cross  used  this  approach  to  determine  the 
impact  of  specific  earned  value  management  (EVM)  system  variables  on  the  growth  of 
DOD  acquisition  programs  (Cross,  2006). 

This  series  of  theses  coded  individual  programs  or  contracts  as  having  cost  or 
schedule  growth  (1)  or  not  having  growth  (0).  These  theses  used  a  logistic  regression 
(logit)  to  determine  which  variables  caused  cost  or  schedule  growth.  The  second  step 
utilized  those  variables  found  to  be  statistically  significant  by  the  logit  as  the  RHS 
variables  in  an  ordinary  least  squares  regression  to  quantify  their  impact  on  cost  or 
schedule  growth  (Cross,  2006). 

Binary  Choice  Models 

The  logistic  regression  used  by  Sipple  is  a  binary  choice  model.  A  binary  choice 
model  is  simply  a  model  with  only  two  possible  outcomes  (Long,  1997).  Mathematically 
it  is  expressed  by  the  relationship 

p,  =  Pr(n  =  i)  (i) 
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(Escudero,  2002).  According  to  Long,  there  are  four  potential  models  for  phenomena  in 
which  two  outcomes  are  possible:  a  linear  probability  model  (LPM)1,  a  binary  logit,  a 
binary  probit,  and  a  complementary  log-log  model  (Long,  1997).  The  last  model  is  not 
addressed  in  this  paper. 


Binary  Logit 

Since  the  LPM  is  not  an  appropriate  estimator  for  a  binary  choice,  a  nonlinear 
model  is  more  appropriate.  A  binary  logit  is  the  first  nonlinear  model  that  will  be 
evaluated.  In  order  to  derive  a  nonlinear  model, 

Pr(y  =  l|3c)  (2) 

must  be  transformed  into  a  function  that  ranges  from  (•  oo,oo) .  The  transfonnation  is 
accomplished  by  setting  1  =  limx;/;  — »  oo  and  0  =  lim  — »  -qo  .  The  first  of  two 


common  cumulative  distribution  functions  that  accommodate  this  transformation  is  the 
standard  logistic  distribution  where 


aL  #)-  “iTA 

l  +  expfc/?) 

Transforming  this  function  into  a  log-likelihood  equation  yields  the  function 

i(/3\y,x)=  Yj  [.A  ln  a(x,'/?)+  0  -  v,  )(ln  A('  A  A))] 


(3) 


(4) 


(Gordon,  2003).  This  function  is  known  as  the  logit. 


1  Addressed  in  Appendix  B. 
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Binary  Probit 


The  second  cumulative  distribution  function  that  utilizes  the  limits 
1  =  lim  — »  oo  and  0  =  lim  — »  -oo  ,  is  the  standard  normal  distribution,  where 


(Milton,  2003).  Since  the  standard  normal  distribution  is  symmetric  about  the  origin  the 
log-likelihood  function  becomes 

*)  =  E [v,  In ©(*;/?)+  (1  - y, (ln®( x,f)%  (6) 

i= 1 

(Gordon,  2003).  This  function  is  known  as  the  probit. 

Binary  Choice  Model  Selection 

The  log-likelihood  functions  for  the  probit  and  the  logit,  are  remarkably  similar. 
Long  and  Escudero  agree  that  both  models  yield  consistent  results.  According  to 
Escudero,  the  only  difference  between  the  distributions  lies  in  the  tails.  This  is  because 
the  coefficients  of  the  independent  variables  are  scaled  differently  (Escudero,  2003).  The 
variance  of  the  probit  is  that  of  the  standard  nonnal  distribution,  or  one,  while  the 

variance  of  the  logit  is  equal  to  71  A  ■  As  such,  the  logit  has  wider  tails  with 

correspondingly  larger  coefficients  (Escudero,  2003). 

Further,  both  the  logit  and  probit  yield  globally  concave  results  (Gordon,  2003). 
Since  the  two  nonlinear  models  are  similar,  “the  choice  between  the  logit  and  probit 
models  is  largely  one  of  convenience  and  convention,  since  the  results  are  generally 
indistinguishable”  (Long,  1997).  Therefore,  the  probit  will  be  utilized  in  this  analysis. 
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Estimation  of  Independent  Variable  Impact  on  Overrun  Magnitudes 

The  second  stage  of  this  study  is  to  estimate  the  impact  of  the  independent 
variables  on  the  magnitude  of  a  cost  overrun  when  one  occurs.  Cost  overruns  can  occur 
for  many  reasons  and  the  range  of  overruns  varies  between  a  negative  value,  a  cost 
underrun,  and  many  times  the  original  value  of  the  contract.  An  OLS  regression  is  the 
simplest  method  to  estimate  the  independent  variables  effects  on  overrun  magnitudes 
(Marsh,  2000).  Therefore,  it  will  be  utilized  to  estimate  the  RHS  variables  impact  on  the 
magnitude  of  cost  overruns. 

Model  Revisited 

A  two  stage  process  will  be  used  to  determine  if  the  independent  variables  of  this 
study  influence  the  likelihood  of  a  contract  experiencing  a  cost  overrun  and  if 
encountered  the  magnitude  of  cost  overrun.  The  first  stage  will  utilize  a  binary  choice 
probit  to  detennine  if  a  contract  will  experience  a  cost  overrun.  The  second  stage  of  this 
study  will  utilize  an  OLS  to  estimate  the  amount  a  contract  will  exceed  its  CBB. 
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IV.  Data  Reduction  and  Results 


Road  Map 

This  chapter  will  combine  the  theory  explored  in  Chapter  II  with  the  methodology 
proposed  in  Chapter  III  to  evaluate  the  impact  of  the  independent  variables  on  the 
likelihood  of  a  cost  overrun  occurring  and  magnitude  of  a  realized  cost  overrun.  The  first 
portion  of  this  chapter  lists  the  sources  of  data  and  explains  how  the  data  was 
manipulated  into  a  usable  fonnat  while  the  second  portion  presents  the  results. 

Data  Collection 

The  data  used  in  this  study  has  been  taken  from  four  sources.  All  contract 
execution  information  has  been  extracted  from  the  Defense  Acquisition  Executive 
Summary  (DAES).  The  changes  in  DOD  procurement  and  RDT&E  budgets  were  culled 
from  the  DOD  Fiscal  Year  2006  Budget  Green  Book.  Industry  concentration  data  was 
pulled  from  the  US  Census  Bureau.  Finally,  inflationary  data  has  been  interpreted  from 
the  Fiscal  Year  2007  President’s  Budget. 

Contract  Information 

The  DAES  database  contains  progress  reports  from  the  individual  services  on  all 
Acquisition  Category  I  programs  (ACAT  1)  (OSD,  2006).  The  services  provide  quarterly 
progress  reports  to  OSD  for  each  active  ACAT  I  program.  The  DAES  database  used  for 
this  analysis  was  originally  used  by  Smirnoff  his  2006  thesis.  Has  database  contained 
14,002  individual  submissions  for  1,150  contracts  that  were  managed  by  the  DOD 
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between  1977  and  2002  (Smirnoff,  2006).  The  Smirnoff  database  contains  a  multitude  of 
contract  information.  However,  this  analysis  will  only  utilize  six  of  his  independent 
variables.  The  variables  used  in  this  analysis  were:  ACWP,  BCWP,  budget  at 
completion  (BAC),  CBB,  submission  date,  and  contract  type. 

Since  most  of  the  progress  reports  in  the  database  were  interim  reports,  the  first 
step  in  the  analysis  was  to  find  the  first  and  last  entry  for  each  contract.  Once  this  was 
accomplished  the  CBBI  was  defined  as  the  CBB  associated  with  the  first  entry  for  each 
contract.  This  value  was  recorded  for  later  use  in  the  analysis.  The  second  step  was  to 
count  the  number  of  times  the  CBB  was  changed  throughout  the  life  of  the  contract.  This 
value  represents  the  number  of  times  the  contract  was  rebaselined. 

A  contract’s  length  was  detennined  by  reviewing  the  submission  date  of  each 
report.  The  date  of  the  first  contract  entry  was  established  as  the  starting  point.  The 
minimum  program  length  was  set  to  one  year.  A  logical  fonnula  was  then  inserted  into 
Microsoft  Excel  to  compare  each  subsequent  submission  date  with  the  preceding  value. 

If  the  year  of  the  more  recent  report  was  greater  than  the  previous  report,  then  the 
program  length  was  increased  by  one  year.  This  method  does  not  round  to  the  nearest 
value,  nor  does  it  allow  for  fractional  values,  however,  it  is  functionally  easy  to  calculate. 

The  key  step  in  the  analysis  was  to  detennine  if  the  contract  exceeded  its  initial 
budget.  This  was  done  by  calculating  the  EAC  and  comparing  this  value  to  the  CBBI.  If 
the  CBBI  was  greater  than  the  EAC,  the  contract  had  not  experienced  a  cost  overrun.  If 
the  CBBI  was  less  than  the  EAC,  the  contract  had  overrun  its  initial  budget.  Since  the 
DAES  only  reports  top-level  data,  the  most  basic  EAC  formula  was  used: 
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EAC  =  A CWP  +  (BAC-  BCWP )  (7) 

(DAU,  2006).  The  magnitude  of  a  cost  overrun  or  underrun  was  determined  by  taking 
the  difference  between  the  EAC  and  the  CBBI. 

Technology  Readiness 

Technology  readiness  was  the  most  difficult  right  hand  side  variable  to  calculate 
with  the  available  data.  The  DAES  database  does  not  provide  technology  readiness 
levels  similar  to  those  used  by  NASA  or  the  Air  Force  Research  Laboratory.  Therefore, 
the  type  of  contract  used  in  the  acquisition  process  was  used  to  proxy  the  technology 
readiness  of  the  commodity  purchased.  The  government  utilizes  cost  plus  contracts  to 
minimize  contractor  risk  for  less  well-defined  technology  and  linn  price  contracts  for  less 
risky  ventures  (Sheffrin,  1976).  In  this  case,  if  the  contract  is  a  fixed  price  contract  or 
any  of  the  fixed  price  variants,  it  is  denoted  with  a  zero;  while  a  cost  plus  contract  is 
represented  with  a  one.  This  structure  is  chosen  to  yield  a  positive  coefficient  for 
technologically  immature  commodities. 

POD  Budgets 

All  DOD  budget  information  was  extracted  from  the  National  Defense  Budget 
Estimates  for  the  Fiscal  Year  2007  (FY07)  Budget,  more  commonly  referred  to  as  the 
FY 07  Green  Book.  All  procurement  and  research  &  development  budget  information  is 
contained  in  Table  6.1  (DOD,  2006).  This  table  lists  the  change  in  real  value,  in  constant 
2007  dollars,  of  DOD  Total  Obligation  Authority,  for  all  appropriation  categories,  as  a 
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percentage  from  the  previous  year.  The  appropriation  categories  that  are  pertinent  in  this 


study  are  the  Procurement  and  RDT&E  appropriations. 


- Procurement 

- RDT&E 


Figure  1:  Percent  Change  of  DOD  Appropriations  (FY77-FY02) 

Industry  Concentration 

All  industry  concentration  data  was  taken  from  surveys  performed  by  the  U.S. 
Census  Bureau.  The  Census  Bureau  publishes  a  census  of  American  industry  every  five 
years  (U.S.  Census,  2006).  These  surveys  are  the  Economic  Census,  Manufacturing 
Surveys.  An  Industry  Concentration  Report  is  included  in  each  Economic  Census, 
Manufacturing  Survey.  All  industry  concentration  data  was  culled  from  the  2002,  1997, 
1992,  1987,  1982,  and  1977  Industry  Concentration  Ratio  reports. 

The  U.S.  Census  Bureau  divides  industry  into  sectors  in  order  to  calculate 
industry  concentration  rations.  The  U.S.  Census  Bureau  used  the  North  American 
Industry  Classification  System  Codes  (NAICS)  to  segment  industry  during  the  1997  and 


19 


2002  censuses.  Prior  to  1997,  the  Census  used  the  Standard  Industrial  Classification 
System  (SIC)  to  divide  American  Industry.  The  Census  Bureau  published  the  1997 
Economic  Census:  Bridge  Between  NAIC  and  SIC.  This  document  provides  a 
crosswalk  to  link  SIC  codes  to  their  corresponding  NAICS  codes.  The  manufacturing 
industry  produces  the  products  that  the  DOD  is  purchasing  with  the  contracts  contained 
in  the  DAES.  The  specific  NAIC  and  SIC  codes  used  in  this  analysis  are: 


Table  1:  NAIC  and  SIC  Crosswalk 


NAIC 

SIC 

Description 

33422 

3663 

Wireless  Communications  Equipement 

33661 

3732 

Ship  and  Boat  Building 

332995 

3489 

Other  Ordnance 

334511 

3812 

Search,  Detection,  Navigation,  Guidance 

336411 

3721 

Aircraft  Manufacturing 

336412 

3724 

Aircraft  Engine 

336413 

3728 

Other  Aircraft  Manufacturing 

336414 

3761 

Guided  Missile/Space  Manufacturing 

336415 

3764 

Guided  Missile/Space  Propulsion 

336419 

3769 

Other  Guided  Missile/Space 

336992 

3795 

Military  Armored  Vehicles 

Source:  U.S.  Bureau  of  the  Census 

The  Industry  Concentration  Ratio  reports  provide  the  number  of  firms  producing 


in  that  given  segment  of  industry;  the  value  of  all  shipments;  the  CR4,  CR8,  CR20,  and 


CR50;  as  well  as  the  HHI.  This  analysis  only  studies  the  impact  of  the  number  of  firms, 


the  CR4,  and  the  CR8.  Since  the  US  Census  Bureau  only  provides  information  once 
every  five  years,  the  data  was  extrapolated  from  one  survey  to  the  next.' 


This  method  of  extrapolation  was  adequate  for  all  but  two  industry  segments. 


These  two  industry  segments  were  the  wireless  communications  industry  and  the  Search, 


Detection,  Navigation,  and  Guidance  industry.  Additionally,  this  method  was 
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problematic  for  the  HHI  as  well.  The  Census  Bureau  masks  the  HHI  for  highly 
concentrated  industries  to  protect  proprietary  infonnation.  This  practice  generated 
several  gaps  in  the  data  for  individual  industry  segments  making  analysis  difficult. 
Therefore,  the  HHI  was  not  used  as  an  independent  variable  in  the  analysis.  The  CR4 
was  selected  as  the  appropriate  measure  to  represent  industry  concentration. 

Inflation 

The  Gross  Domestic  Product  (GDP)  implicit  price  deflator  (IPD)  is  considered  to 
be,  “the  single  best  measure  of  broad  price  movements  in  the  economy”  (DOD,  2007). 
The  IPD  is  the  ratio  of  “GDP  in  current  prices  to  the  GDP  in  constant  prices”  (DOD, 
2007).  Therefore  the  IPD  will  be  used  to  calculate  a  proxy  for  inflation.  It  is  expressed 
as  a  chain  index  where  the  base  year  is  equal  to  1.00.  The  GDP  chain  index  was 
extracted  from  the  FY07  Budget  of  the  United  States  Government,  Table  10.1.  The  base 
year  for  the  index  in  this  table  is  2000.  The  growth  in  the  GDP  chain-index,  over  the 
period  of  analysis,  is  presented  below: 


2  All  industry  concentration  is  provided  in  Appendix  A. 
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Figure  2:  Growth  in  GDP  (1977-2006) 

Inflation  is  calculated  by  dividing  the  most  current  year  GDP  chain  index  by  the 
chain  index  of  the  year  in  question  (Davis,  2006).  For  example,  inflation  for  the  year 
1977  was  found  to  be  2.71  percent.  This  value  was  found  by  dividing  the  2006  chain 
index  value  of  1.1475  by  0.4233,  the  1977  chain  index.  Inflation,  over  the  period  of 
analysis  is  presented  below. 


Year 


Figure  3:  Inflation  (1977-2006) 
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Results 


The  RHS  variables  discussed  throughout  this  paper  will  be  evaluated  twice.  The 
first  evaluation  will  look  at  their  influence  on  the  likelihood  of  a  cost  overrun  occurring, 
while  the  second  evaluation  will  determine  their  impact  on  the  magnitude  of  a  realized 
cost  overrun. 

Summary  Statistics 

The  summary  statistics  for  the  data  used  in  this  analysis  are: 
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Table  2:  Summary  Statistics 


Variable 

obs 

Mean 

Std.  Dev. 

Min 

Max 

Overrun 

1150 

0.7095652 

0.4541605 

0 

1 

Magnitude 

1150 

102.1169 

603.7141 

-1493.6 

10673.7 

ln(Magnitude  Cost  Overrun) 

818 

3.283981 

1.860017 

-2.302585 

9.275538 

ln(Percent  Cost  Overrun) 

1149 

0.7274253 

4.663351 

-0.6804309 

143.5 

GDP  Deflator 

1150 

1.522135 

0.3603345 

1.1 

2.7108 

Change  Procurement  Budget 

1150 

-0.4854783 

11.31916 

-19.9 

24.5 

Change  RDT&E  Budget 

1150 

3.278869 

7.373217 

-10.2 

14.9 

CR4 

1150 

61.41043 

19.62887 

11 

96 

Number  Rebaselines 

1150 

6.733043 

6.55399 

0 

39 

Contract  Length 

1150 

3.641739 

2.049092 

0 

14 

Technology  Readiness 

1150 

0.4052174 

0.4911477 

0 

1 

The  data  indicates  that  nearly  7 1  percent  of  the  DOD  contracts  encountered  cost  overruns 


with  an  average  amount  of  $  102  million.  However,  this  number  is  influenced  by  four 


contracts  for  the  C-17,  A-12  and  Trident  submarine  that  accounted  for  nearly  $9.7B.  All 


contracts  averaged  3.6  years  with  nearly  seven  rebaselines  each.  Further,  inflation 


averaged  1.5  percent  annually,  with  an  average  CR4  of  61  percent.  Finally,  40  percent 


were  fixed  price  contracts  while  60  percent  were  cost-plus  contracts. 


Binary  Choice 

The  model  utilized  to  explore  the  likelihood  of  a  cost  overrun  occurring  is  a  probit 

where: 

Pr(CO)  =  f(GDP,  APcmt,  A R  &  D ,# BaselineChanges ,  Length, TR,  CR4)  (8) 

In  this  case  GDP  is  equal  to  the  inflation  derived  from  the  GDP  IPD,  APcmt  is  the  annual 
percent  change  in  the  DOD  Procurement  budget,  AR&D  is  the  annual  percent  change  in 
the  RDT&E  budget,  #BaselineChanges  is  the  number  of  times  the  CBB  has  changed, 
Length  is  the  length  of  contract,  TR  is  the  technology  readiness  proxy,  and  CR4  is  the 
industry  CR4  for  a  given  contract. 
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The  probit  model  yielded  the  following  results  when  reporting  the  change  in 


probability: 


Table  3:  Probit  Model  Results 


Variable 

dF/dx 

P(Z) 

GDP  Deflator 

0.0621359 

0.149 

Change  in  Procurement 

0.0011605 

0.485 

Change  in  R&D 

0.0006480 

0.782 

CR4 

0.0001576 

0.825 

Number  Rebaselines*** 

0.0201413 

0.000 

Contract  Length*** 

0.0371733 

0.002 

Technology  Readiness 

0.0356250 

0.196 

Obs  P. 

0.7095652 

Pred  P. 

0.7460646  at  x-bar 

Pseudo  R2 

0.1117 

P(Chi) 

0.0000 

Observations 

1150 

***  Significant  to  the  1%  level 
**  Significant  to  the  5%  level 
*  Significant  to  the  10%  level 


The  marginal  effects  value,  or  DF/dx,  represents  the  discrete  change  of  the  dependent 
variable  from  a  zero  to  one.  In  this  case  the  discrete  change  occurs  when  a  contract 
overruns  its  CBBI.  The  pseudo-R  of .  1 1 1 7  indicates  that  contract  management  is  a 
volatile  process  and  the  likelihood  of  a  contact  experiencing  a  cost  overrun  is  highly 
stochastic.  Despite  the  fact  that  nearly  89  percent  of  the  causes  of  a  cost  overrun  are  not 
explained  by  this  model  the  high  p-score  indicates  that  the  model  has  tremendous 
statistical  significance. 

Only  two  of  the  independent  variables  used  to  analyze  the  occurrence  of  cost 
overruns  proved  to  be  statistically  significant.  These  two  variables  were  the  number  of 
rebaselines  and  the  length  of  the  contract.  However,  both  variables  were  statistically 


’  The  probit  is  presented  here  as  it  is  consistent  with  the  models  used  in  similar  studies  in  the  economic 
literature.  Since  the  logit  model  is  more  commonly  found  it  cost  research  it  is  presented  along  with  a  linear 
probability  model  in  Appendix  B.  All  three  models  yielded  similar  results. 
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significant  to  the  one  percent  level,  indicating  a  very  powerful  positive  influence  on  the 
likelihood  of  an  overrun  occurring.  As  such,  contract  budget  instability  and  extending 
the  length  of  a  contract  both  add  to  the  likelihood  of  a  contract  experiencing  a  cost 
overrun.  Each  time  a  contract  is  rebaselined  it  is  two  percent  more  likely  to  experience  a 
cost  overrun  while  an  additional  year  in  program  length  adds  3.6  percent  to  the  likelihood 
of  a  cost  overrun.  The  remaining  variables  had  no  influence  on  the  likelihood  of  a 
contract  overrunning  its  CBBI.4 

A  Hausman  test  was  performed  to  determine  if  there  are  endogenous  variables  in 
the  model.  The  Hausman  test  concluded  that  none  of  the  RHS  variables  were 
endogenous.5  Therefore,  contract  budget  instability  and  contract  length  are  accurate 
predictors  of  cost  overruns. 

Impact  on  Magnitude 

The  left-hand  side  variable  used  in  OLS  regression  was  the  magnitude  of  the  cost 
overrun.  This  number  calculated  by  taking  the  difference  between  the  most  recent  EAC 
and  the  CBBI.  The  specific  model  used  was: 

OverrunMagnitude  =  f(AGDP,  APcmt,  A R  &  D,  InitialCRA,# BaselineChanges ,  Length,  TR,CBBl) 

(9) 

The  RHS  variables  used  in  this  model  are  largely  the  same  as  the  variables  used  in  the 
probit  model.  The  CBBI  was  added  to  help  scale  the  magnitude  while  the  change  in  GDP 
IPD  was  used  to  measure  the  effects  of  inflation  over  the  life  of  the  contract.  The  CR4 

4  The  model  proved  highly  insensitive  to  the  measure  selected  to  represent  industry  concentration.  All 
regressions  provided  very  similar  results.  Additional  models  utilizing  alternate  measures  of  industry 
concentration  are  contained  in  Appendix  C. 

5  Post  estimation  results  are  included  in  Appendix  B. 
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encountered  when  the  contract  was  signed  was  used  to  measure  industry  concentration 


with  regard  to  the  initial  contract  price.  The  regression  results  are: 


Table  4:  OLS  Model  Results 


Variable 

Coefficient 

P(t) 

Change  in  GDP  Deflator 

147.145700 

0.2600 

Change  in  Procurement 

0.267741 

0.9010 

Change  in  R&D 

2.460645 

0.4200 

Initial  CR4** 

2.281388 

0.0180 

Number  Rebaselines*** 

27.198500 

0.0000 

Contract  Length** 

43.802330 

0.0040 

Technology  Readiness 

-56.120910 

0.1340 

CBBI** 

0.043070 

0.0140 

Intercept*** 

-335.462200 

0.0000 

Adjusted  R2 

0.1539 

P(F) 

0.0000 

Observations 

1055 

Bruesch-Pagan 

Het  Test 

Chi2(1) 

5738.26 

P(Chi) 

0.0000 

Ramsey  Omitted  Variable  Test 

F(3,  1043) 

121.3 

P(F) 

0.0000 

***  Significant  to  1%  level 
**  Significant  to  5%  level 


The  adjusted  r-squared  value  indicates  that  this  model  explains  nearly  16%  of  the  factors 
that  influence  the  amount  a  contract  over  run  its  initial  budget  while  the  p-score  indicates 
that  the  model  is  very  significant.  In  conjunction  with  the  high  p-score,  five  of  the  nine 
RHS  variables  are  statistically  significant  when  considering  how  much  a  contract  will 
exceed  its  CBBI. 

The  number  of  rebaselines,  and  intercept  are  significant  to  the  one  percent  level 
while  the  initial  CR4,  CBBI,  and  contract  length  are  significant  to  the  five  percent  level. 

6  A  second  OLS  regression  in  which  the  cost  overrun  magnitude  is  a  function  of  change  in  the  GPD  IPD, 
change  in  procurement  budget,  change  in  R&D  budget,  change  in  CR4,  number  of  rebaselines,  contract 
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Consequently,  as  the  CR4  increases  by  one  percent  the  magnitude  of  a  cost  overrun  will 
increase  by  $2.3M.  Likewise  the  magnitude  of  an  overrun  will  be  increased  by  $27.2M 
for  each  rebaseline  and  $43. 8M  for  each  additional  year.  The  CBBI  while  statistically 
significant  has  a  very  small  impact  on  the  amount  of  a  cost  overrun,  although  it  does 
indicate  that  larger  contracts  will  experience  more  cost  overruns.  The  variables  that  were 
not  statistically  significant  to  the  impact  the  magnitude  of  a  cost  overrun  were  the  change 
in  the  GDP  IPD,  changes  in  the  procurement  and  RDT&E  budgets,  and  the  type  of 
contractual  vehicle  used. 

Summary 

The  probit  model  presented  in  this  chapter  provides  significant  insight  into  the 
economic  factors  that  influence  the  likelihood  of  a  contract  experiencing  a  cost  overrun. 

It  indicates  that  rebaselines  and  contract  length  are  prime  indicators  that  a  contract  will 
overrun  its  CBBI.  Additionally,  the  OLS  model  indicates  that  the  number  of  times  a 
contract  is  rebaselined,  contract  length,  and  initial  CR4  have  a  significant  impact  on  the 
magnitude  of  a  cost  overrun. 


length,  technology  readiness,  and  CBBI  yielded  similar  coefficients  and  magnitudes  with  a  adjusted  r- 
squared  of  0.1521.  The  rebaseline  coefficient  was  25.5  and  the  contract  length  coefficient  was  41.2. 
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V.  Implications  and  Conclusion 


Results  Revisited 

The  goal  of  this  research  was  to  analyze  DOD  contracts  to  detennine  if 
fluctuating  contract  budgets  and  macroeconomic  variables  influence  the  likelihood  of  a 
cost  overrun  for  DOD  acquisition  contracts,  and  if  so  by  how  much.  The  analysis  utilized 
a  two-step  process  to  determine  if  the  likelihood  and  magnitude  of  cost  overruns  as  a 
function  of  inflation,  the  number  of  contract  rebaselines,  change  in  procurement  budget, 
change  in  RDT&E  budget,  CR4,  contract  length,  and  TR  of  the  purchased  commodity. 
The  analysis  concluded  that  the  likelihood  of  a  cost  overrun  is  influenced  by  both  the 
number  of  times  the  contract  is  rebaselined  and  the  program  length.  Each  time  an 
acquisition  contract  is  rebaselined  the  likelihood  of  a  cost  overrun  occurring  increased  by 
two  percent.  Further,  each  year  a  contract  is  delayed  the  likelihood  of  a  cost  overrun  is 
increased  by  3.6  percent. 

The  analysis  also  determined  that  the  magnitude  of  a  cost  overrun  is  influenced  by 
the  number  of  rebaselines,  initial  CR4,  contract  length,  and  CBBI  of  the  commodity. 
There  were  an  average  of  44  active  contracts  each  year  between  1977  and  2002  and  71% 
of  these  contracts  final  costs  exceeded  their  CBBI.  This  translated  into  nearly  $850M  in 
cost  overruns  a  year  if  each  contract  experienced  a  rebaseline,  or  $183M  per  contract 
since  the  average  contract  was  rebaselined  6.7  times.  Further,  extending  all  active 
contracts  an  extra  year  adds  $1.4B  to  annual  procurement  costs.  Finally,  as  the  defense 
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industry  becomes  more  concentrated  the  annual  cost  for  all  contracts  is  an  additional 
$71M  for  each  percentage  point  rise  in  the  CR4. 

Policy  Implications 

The  implications  of  these  results  to  the  DOD  are  relatively  straightforward.  A 
major  culture  change  should  be  encouraged  inside  the  DOD.  The  DOD  should  not 
attempt  to  evolve  contracts  as  new  technology  becomes  available.  This  practice 
unnecessarily  lengthens  acquisition  programs  and  drives  up  costs  and  encourages  budget 
instability  for  individual  contracts.  Program  managers  should  be  encouraged  to  keep 
acquisition  programs  short  and  resist  the  temptation  to  buy  the  most  technologically 
advanced  systems  possible.  Technology  should  be  incorporated  into  existing  weapons 
systems  via  small  upgrades  utilizing  the  spiral  acquisition  model.  This  practice  will 
make  the  goals  of  acquisition  programs  more  realistic  and  curb  the  ever  lengthening  life 
span  of  DOD  acquisition  programs. 

A  second  major  implication  of  this  study  is  to  the  defense  marketplace.  The 
defense  industry  consolidation  that  has  been  encouraged  over  the  past  two  decades 
should  be  halted  and  reversed.  Although  significant  barriers  exist  to  entry  in  the  defense 
industry  as  a  prime  contractor,  this  analysis  was  not  limited  to  primes.  The  DOD  should 
encourage  more  competition  in  the  subcontractor  arena  which  should  in  turn  lead  to  more 
prime  contractors  in  the  future.  The  combination  of  these  two  proposals,  constraining  the 
technology  procurement  of  ongoing  acquisition  programs  and  halting  the  defense 
industry  consolidation  should  reduce  acquisition  costs  by  several  million  dollars  for  each 
major  acquisition  program. 
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Suggestions  for  Further  Research 


This  research  can  be  extended  in  several  ways.  Probably  the  most  useful 
extension  would  be  to  determine  what  proportion  of  contract  cost  overruns  are  caused  by 
changing  requirements  and  the  proportion  generated  by  poor  cost  estimates.  A  second 
way  to  extend  this  research  would  be  to  apply  this  model  to  the  individual  services  and 
different  types  of  systems  procured  by  the  DOD.  Finally,  the  technology  readiness  proxy 
utilized  by  this  study  is  very  rudimentary.  Enhancing  the  technology  readiness  level 
proxy  may  provide  more  powerful  insight  into  the  causes  of  contract  cost  overruns. 
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Appendix  A:  NAIC  and  SIC  Information 


This  Appendix  lists  the  industry  concentration  information  extracted  from  the  US 
Census  Bureau’s  Survey  of  Industry,  Industry  Concentration  Report.  Each  table  contains 
the  extrapolated  data  for  a  single  NAIC/SIC  segment  of  the  manufacturing  industry. 


Table  5:  33422  Wireless  Communications  Equipment 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

934 

44 

55 

584 

2001 

966 

45 

56 

661 

2000 

999 

46 

57 

739 

1999 

1031 

47 

58 

817 

1998 

1064 

48 

58 

894 

1997 

1096 

49 

59 

972 

1996 

1049 

47 

58 

927 

1995 

1002 

46 

57 

883 

1994 

955 

44 

55 

838 

1993 

908 

43 

54 

794 

1992 

861 

41 

53 

749 

1991 

803 

40 

53 

700 

1990 

745 

39 

54 

650 

1989 

688 

39 

54 

601 

1988 

630 

38 

55 

551 

1987 

572 

37 

55 

502 

1986 

572 

37 

55 

502 

1985 

572 

37 

55 

502 

1984 

572 

37 

55 

502 

1983 

572 

37 

55 

502 

1982 

572 

37 

55 

502 

1981 

572 

37 

55 

502 

1980 

572 

37 

55 

502 

1979 

572 

37 

55 

502 

1978 

572 

37 

55 

502 

1977 

572 

37 

55 

502 

The  SIC  code  for  the  wireless  communications  industry  was  established  with  the  1987 
industry  census. 
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Table  6:  33611  Ship  and  Boat  Building 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

1649 

51 

56 

884 

2001 

1645 

48 

55 

797 

2000 

1641 

46 

54 

710 

1999 

1638 

43 

53 

623 

1998 

1634 

41 

52 

536 

1997 

1630 

38 

51 

450 

1996 

1779 

37 

48 

436 

1995 

1928 

36 

46 

423 

1994 

2078 

35 

43 

410 

1993 

2227 

33 

41 

396 

1992 

2376 

32 

38 

383 

1991 

2322 

32 

39 

416 

1990 

2269 

32 

39 

449 

1989 

2215 

33 

40 

482 

1988 

2162 

33 

40 

515 

1987 

2108 

33 

41 

548 

1986 

2053 

29 

37 

457 

1985 

1998 

25 

33 

365 

1984 

1944 

22 

30 

274 

1983 

1889 

18 

26 

182 

1982 

1834 

14 

22 

91 

1981 

1897 

13 

21 

73 

1980 

1960 

13 

21 

55 

1979 

2022 

12 

20 

36 

1978 

2085 

12 

20 

18 

1977 

2148 

11 

19 

0 
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Table  7:  332995  Other  Ordnance 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

58 

78 

90 

2994 

2001 

59 

79 

90 

2811 

2000 

61 

79 

90 

2628 

1999 

62 

80 

90 

2445 

1998 

64 

81 

89 

2262 

1997 

65 

82 

89 

2080 

1996 

66 

82 

89 

2050 

1995 

67 

82 

89 

2019 

1994 

69 

83 

89 

1989 

1993 

70 

83 

89 

1959 

1992 

71 

83 

89 

1929 

1991 

69 

82 

89 

1915 

1990 

66 

81 

89 

1901 

1989 

64 

79 

90 

1886 

1988 

61 

78 

90 

1872 

1987 

59 

77 

90 

1858 

1986 

60 

73 

88 

1699 

1985 

61 

70 

87 

1539 

1984 

63 

66 

85 

1380 

1983 

64 

63 

84 

1220 

1982 

65 

59 

82 

1061 

1981 

70 

57 

80 

849 

1980 

75 

55 

78 

637 

1979 

79 

52 

76 

424 

1978 

84 

50 

74 

212 

1977 

89 

48 

72 

0 
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Table  8:  334511  Search,  Detection,  Navigation,  Guidance 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

504 

59 

77 

1196 

2001 

519 

59 

77 

1203 

2000 

534 

59 

77 

1211 

1999 

549 

59 

77 

1219 

1998 

564 

59 

77 

1227 

1997 

579 

59 

77 

1235 

1996 

590 

53 

71 

1065 

1995 

601 

46 

65 

895 

1994 

613 

40 

60 

725 

1993 

624 

33 

54 

555 

1992 

635 

27 

48 

385 

1991 

692 

27 

48 

388 

1990 

748 

28 

48 

391 

1989 

805 

28 

49 

395 

1988 

861 

29 

49 

398 

1987 

918 

29 

49 

401 

1986 

918 

29 

49 

401 

1985 

918 

29 

49 

401 

1984 

918 

29 

49 

401 

1983 

918 

29 

49 

401 

1982 

918 

29 

49 

401 

1981 

918 

29 

49 

401 

1980 

918 

29 

49 

401 

1979 

918 

29 

49 

401 

1978 

918 

29 

49 

401 

1977 

918 

29 

49 

401 

The  SIC  code  for  the  search,  detection,  navigation  and  guidance  industry  was  established 
with  the  1987  industry  census. 
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Table  9:  336411  Aircraft  Manufacturing 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

184 

81 

94 

2948 

2001 

182 

81 

94 

2863 

2000 

179 

81 

94 

2779 

1999 

177 

81 

94 

2695 

1998 

174 

81 

94 

2610 

1997 

172 

81 

95 

2526 

1996 

168 

81 

94 

2564 

1995 

164 

80 

94 

2603 

1994 

159 

80 

94 

2641 

1993 

155 

79 

93 

2679 

1992 

151 

79 

93 

2717 

1991 

148 

78 

93 

2547 

1990 

145 

76 

93 

2377 

1989 

143 

75 

92 

2208 

1988 

140 

73 

92 

2038 

1987 

137 

72 

92 

1868 

1986 

137 

70 

90 

1766 

1985 

138 

69 

88 

1664 

1984 

138 

67 

85 

1562 

1983 

139 

66 

83 

1460 

1982 

139 

64 

81 

1358 

1981 

141 

63 

81 

1086 

1980 

144 

62 

81 

815 

1979 

146 

61 

81 

543 

1978 

149 

60 

81 

272 

1977 

151 

59 

81 

0 
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Table  10:  336412  Aircraft  Engine 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

296 

77 

82 

2102 

2001 

293 

76 

82 

2033 

2000 

290 

76 

81 

1963 

1999 

287 

75 

81 

1893 

1998 

284 

75 

80 

1823 

1997 

281 

74 

80 

1754 

1996 

293 

75 

81 

1879 

1995 

305 

75 

82 

2003 

1994 

316 

76 

82 

2128 

1993 

328 

76 

83 

2253 

1992 

340 

77 

84 

2378 

1991 

346 

77 

84 

2343 

1990 

353 

77 

84 

2307 

1989 

359 

77 

83 

2272 

1988 

366 

77 

83 

2236 

1987 

372 

77 

83 

2201 

1986 

354 

76 

83 

2116 

1985 

336 

75 

83 

2032 

1984 

317 

74 

83 

1947 

1983 

299 

73 

83 

1863 

1982 

281 

72 

83 

1778 

1981 

270 

72 

84 

1422 

1980 

259 

73 

84 

1067 

1979 

248 

73 

85 

711 

1978 

237 

74 

85 

356 

1977 

226 

74 

86 

0 
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Table  11:  336413  Other  Aircraft  Manufacturing 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

760 

54 

69 

950 

2001 

818 

54 

69 

985 

2000 

876 

54 

69 

1020 

1999 

933 

55 

69 

1056 

1998 

991 

55 

69 

1091 

1997 

1049 

55 

69 

1126 

1996 

1045 

53 

67 

1055 

1995 

1041 

51 

66 

985 

1994 

1036 

48 

65 

914 

1993 

1032 

46 

63 

843 

1992 

1028 

44 

62 

772 

1991 

1007 

44 

61 

748 

1990 

987 

43 

60 

724 

1989 

966 

43 

59 

700 

1988 

946 

42 

58 

676 

1987 

925 

42 

57 

652 

1986 

922 

41 

56 

641 

1985 

920 

40 

56 

630 

1984 

917 

40 

55 

620 

1983 

915 

39 

55 

609 

1982 

912 

38 

54 

598 

1981 

866 

39 

54 

478 

1980 

820 

41 

55 

359 

1979 

773 

42 

55 

239 

1978 

727 

44 

56 

120 

1977 

681 

45 

56 

0 
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Table  12:  336414  Guided  Missile/Space  Manufacturing 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

13 

96.0 

99.0  D 

2001 

13 

95.1 

99.1 

D 

2000 

14 

94.2 

99.2 

D 

1999 

14 

93.4 

99.4 

D 

1998 

15 

92.5 

99.5 

D 

1997 

15 

91.6 

99.6 

D 

1996 

17 

87 

98 

D 

1995 

19 

83 

97 

D 

1994 

20 

79 

96 

D 

1993 

22 

75 

94 

D 

1992 

24 

71 

93 

1570 

1991 

23 

68 

93 

1500 

1990 

22 

66 

93 

1430 

1989 

21 

63 

92 

1360 

1988 

20 

61 

92 

1290 

1987 

19 

58 

92 

1220 

1986 

18 

61 

93 

1292 

1985 

18 

63 

94 

1363 

1984 

17 

66 

94 

1435 

1983 

17 

68 

95 

1506 

1982 

16 

71 

96 

1578 

1981 

17 

70 

96 

1262 

1980 

18 

68 

95 

947 

1979 

18 

67 

95 

631 

1978 

19 

65 

94 

316 

1977 

20 

64 

94 

0 
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Table  13:  336415  Guided  Missile/Space  Propulsion 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

16 

90 

99  D 

2001 

17 

88 

99  D 

2000 

17 

86 

98  D 

1999 

18 

83 

98  D 

1998 

18 

81 

98  D 

1997 

19 

79 

97 

2056 

1996 

21 

77 

96 

1934 

1995 

23 

76 

95 

1812 

1994 

24 

74 

94 

1690 

1993 

26 

73 

93 

1568 

1992 

28 

71 

92 

1446 

1991 

28 

71 

93 

1471 

1990 

28 

72 

94 

1496 

1989 

27 

72 

95 

1520 

1988 

27 

73 

96 

1545 

1987 

27 

73 

97 

1570 

1986 

26 

72 

96 

1536 

1985 

24 

71 

95 

1503 

1984 

23 

70 

95 

1469 

1983 

21 

69 

94 

1436 

1982 

20 

68 

93 

1402 

1981 

20 

68 

93 

1122 

1980 

19 

68 

93 

841 

1979 

19 

69 

93 

561 

1978 

18 

69 

93 

280 

1977 

18 

69 

93 

0 
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Table  14:  336419  Other  Guided  Missile/Space 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

51 

60 

75 

2515 

2001 

50 

63 

77 

2478 

2000 

50 

65 

79 

2440 

1999 

49 

67 

81 

2403 

1998 

49 

69 

84 

2365 

1997 

48 

72 

86 

2327 

1996 

49 

72 

86 

2269 

1995 

50 

73 

86 

2210 

1994 

52 

74 

86 

2151 

1993 

53 

74 

86 

2093 

1992 

54 

75 

86 

2034 

1991 

55 

72 

84 

1897 

1990 

57 

70 

83 

1760 

1989 

58 

67 

81 

1624 

1988 

60 

65 

80 

1487 

1987 

61 

62 

78 

1350 

1986 

58 

50 

62 

1080 

1985 

55 

37 

47 

810 

1984 

51 

25 

31 

540 

1983 

48 

12 

16 

270 

1982 

45 

0 

0 

0 

1981 

44 

15 

17 

0 

1980 

43 

30 

34 

0 

1979 

43 

46 

52 

0 

1978 

42 

61 

69 

0 

1977 

41 

76 

86 

0 
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Table  15:  336992  Military  Armored  Vehicles 


Year 

#  Firms 

CR4 

CR8 

HHI 

2002 

31 

88 

93  D 

2001 

32 

88 

93  D 

2000 

33 

87 

93  D 

1999 

35 

87 

93  D 

1998 

36 

87 

93  D 

1997 

37 

86 

93  D 

1996 

37 

86 

93  D 

1995 

37 

87 

93  D 

1994 

37 

87 

93  D 

1993 

37 

88 

94  D 

1992 

37 

88 

94 

2320 

1991 

40 

89 

94 

1856 

1990 

42 

90 

94 

1392 

1989 

45 

90 

95 

928 

1988 

47 

91 

95 

464 

1987 

50 

92 

95 

0 

1986 

48 

91 

93 

503 

1985 

45 

89 

91 

1006 

1984 

43 

88 

89 

1509 

1983 

40 

86 

87 

2012 

1982 

38 

85 

85 

2515 

1981 

34 

85 

87 

2012 

1980 

31 

86 

90 

1509 

1979 

27 

86 

92 

1006 

1978 

24 

87 

95 

503 

1977 

20 

87 

97 

0 
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Appendix  B:  Probit  Post  Estimation  Results  and  Alternate  Models 


This  appendix  presents  the  Hausman  test  post  estimation  results  and  additional 
models  with  utilizing  the  CR4  as  to  measure  industry  concentration.  The  first  model 
presented  is  the  initial  probit  model  contained  in  Chapter  IV,  including  residuals  as  an 
independent  variable.  The  results  indicate  that  the  only  variable  that  is  statistically 
significant  is  the  residual  variable.  This  model  is  then  compared  to  the  initial  model  via 
the  Hausman  test.  The  Hausman  test  revealed  that  the  initial  model  provided  efficient 
coefficient  estimators  while  the  second  model  with  residuals  does  not  provide  efficient 
estimators.  This  indicates  that  endogeneity  is  not  contained  in  the  model. 


Table  16:  Probit  with  Residuals 


Variable 

dF/dx 

P(Z) 

GDP  Deflator 

0.0141148 

0.783 

Change  in  Procurement 

0.0001945 

0.912 

Change  in  R&D 

0.0002687 

0.910 

CR4 

0.0000218 

0.976 

Number  Rebaselines 

0.0089044 

0.229 

Contract  Length 

0.0094943 

0.632 

Technology  Readiness 

0.0118328 

0.703 

Residuals* 

0.7124867 

0.075 

Obs  P. 

0.7095652 

Pred  P. 

0.7401252 

at  x-bar 

Pseudo  R2 

0.1138000 

Log  Likelihood 

-614.08974 

P(Chi) 

0.0000 

Observations 

1150 
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Table  17:  Hausman  Results 


Coefficients 

b  (Before)  B  (after)  b-B  sqrt(diag(v b-v B)) 

GDP  Deflator 

0.1939282 

0.0435259 

0.1504023 

- 

Change  in  Procurement 

0.0036219 

0.0005998 

0.0030221 

- 

Change  in  R&D 

0.0020226 

0.0008286 

0.001194 

- 

CR4 

0.000492 

0.0000671 

0.0004249 

- 

Number  Rebaselines 

0.0628618 

0.0274585 

0.0354033 

- 

Program  Length 

0.1160192 

0.0292778 

0.0867414 

- 

Technology  Readiness 

0.1120096 

0.0365718 

0.0754378 

- 

The  next  two  models  present  alternate  methods  to  estimate  the  likelihood  of  a  cost 
overrun  occurring  on  a  DOD  procurement  contract.  The  first  competing  model  is  the 
logit  while  the  second  is  a  simple  ordinary  least  squares  (OLS)  estimation. 


Table  18:  Logit  Model 


Variable 

Odds  Ratio 

P(Z) 

GDP  Deflator 

1.3335890 

0.190 

Change  in  Procurement 

1.0058870 

0.500 

Change  in  R&D 

1.0015870 

0.897 

CR4 

0.9999807 

0.856 

Number  Rebaselines*** 

1.1275950 

0.000 

Contract  Length*** 

1.1994520 

0.005 

Technology  Readiness 

1.2083360 

0.193 

Pseudo  R2 

0.1133 

Log  Likelihood 

-614.3977 

P(Chi) 

0.0000 

Observations 

1150 

Pearson  Goodness  of  Fit 

Number  Covariate  Patterns 

996 

Pearson  Chi2  (988) 

1030.08 

P(Chi) 

0.1714 

A  LPM  is  merely  an  OLS  regression  to  fit  binary  choice  phenomena.  As  such, 
the  LPM  provides  the  worst  results  of  the  three  models  that  will  be  evaluated.  It  does  this 


because  the  observed  phenomena  produce  an  S-shaped  curve  that  relates  the  probability 
of  an  event  occurring  to  the  studied  independent  variables,  yet  it  linear  relationship  is 
produced  (Long,  1997). 
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This  linear  relationship  is  problematic  for  several  reasons.  The  first  is  that  the 
data  is  heteroskedastic.  The  S-shaped  nature  of  the  curve  indicates  that  the  data  does  not 
have  a  constant  variance,  therefore  a  key  assumption  of  the  OLS  is  violated  and  the 
resulting  coefficients  are  inefficient  estimators  of  probability  (Long,  1997).  The  second 
assumption  violated  when  a  LPM  is  used  to  estimate  a  binary  choice  model  is  that  of 
nonnality  of  the  error  terms.  The  error  terms  must  result  from  either 

e,=\-E(y\x,)  (13) 

or 

sQ=0-E(y\x).  (14) 

Hence,  the  error  terms  can  not  be  normally  distributed.  Although  normally  distributed 
error  tenns  are  preferred,  they  are  not  required  (Long,  1997). 

Finally,  a  LPM  can  produce  results  that  simply  do  not  make  sense.  In  this 
construct  the  expected  value  of  y  given  a  vector  of  independent  variables  is 

£(y|x)=Pr(v  =  l|x)  (15) 

(Long,  1997).  Since  a  binary  choice  model  is  bounded  (0,1),  and  it  is  possible  for 
a  LPM  to  yield  a  result  outside  this  range,  the  LPM  can  produce  results  that  do  not  fit  the 
underlying  data  (Gordon,  2003). 

The  LPM  yields  an  inefficient  estimator  of  Pr(l)  =  l)  in  binary  response 

situations.  It  is  inefficient  for  the  reasons  stated  above:  it  does  not  have  a  common 
variance  and  it  yields  answers  outside  the  possible  range.  Despite  these  shortcomings,  it 
can  provide  a  rough  estimate  before  more  rigorous  analysis  is  conducted. 
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Table  19:  OLS  Model 


Variable  Coefficient  P(t) 

GDP  Deflator 

0.059648  0.146 

Change  in  Procurement 

0.001278 

0.415 

Change  in  R&D 

0.000085 

0.969 

CR4 

-0.000064 

0.925 

Number  Rebaselines*** 

0.013196 

0.000 

Contract  Length*** 

0.034892 

0.001 

Technology  Readiness 

0.030369 

0.252 

Intercept*** 

0.394804  0.000 

Adjusted  R2 

0.1028 

P(F) 

0.0000 

Observations 

1150 

Bruesch-Pagan  Het  Test 

Chi2(1) 

69.01 

P(Chi) 

0.0000 

Ramsey  Omitted  Variable  Test 

F(3,  1139) 

7.221 

P(F) 

0.0001 

The  results  of  these  two  models  are  consistent  with  the  probit  presented  in  the  main  text. 
All  three  models  yield  the  same  significant  variables  and  very  similar  pseudo-  and 


adjusted  r-squared  results.  This  indicates  the  model  is  highly  insensitive  to  functional 


form.  Further,  the  OLS  model  passed  the  Bruesch-Pagan  heteroskedasticity  test  and  the 


Ramsey  Omitted  Variable  test  at  the  one  percent  level. 


46 


Appendix  C:  Alternate  Measures  of  Industry  Concentration 


This  appendix  presents  the  two  additional  probit  models  that  utilize  alternate 
measures  of  industry  concentration  to  determine  the  likelihood  of  a  cost  overrun 


occurring.  The  first  model  utilizes  the  number  of  firms  in  a  given  segment  of  the  defense 
industry  while  the  second  utilizes  the  CR8. 


Table  20:  Probit  (Firms) 


Variable 

dF/dx 

P(Z) 

GDP  Deflator 

0.0589753 

0.165 

Change  in  Procurement 

0.0011540 

0.486 

Change  in  R&D 

0.0006637 

0.776 

Firms 

-0.0000076 

0.701 

Number  Rebaselines*** 

0.0203382 

0.000 

Program  Length*** 

0.0367790 

0.003 

Technology  Readiness 

0.0349616 

0.205 

Obs  P. 

0.7095652 

Pred  P. 

0.7461283  at  x-bar 

Pseudo  R2 

0.1118 

Log  Likelihood 

-615.4402 

P(Chi) 

0.0000 

Observations 

1150] 

Table  21:  Probit  (CR8) 


Variable 

dF/dx 

P(Z) 

GDP  Deflator 

0.0602118 

0.155 

Change  in  Procurement 

0.0012157 

0.462 

Change  in  R&D 

0.0006712 

0.774 

CR8 

-0.0000634 

0.922 

Number  Rebaselines*** 

0.0199631 

0.000 

Program  Length*** 

0.0374885 

0.002 

Technology  Readiness 

0.0364329 

0.186 

Obs  P. 

0.7095652 

Pred  P. 

0.7459790  at  x-bar 

Pseudo  R2 

0.1117 

Log  Likelihood 

-615.5089 

P(Chi) 

0.0000 

Observations 

1150| 
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These  results  are  consistent  with  the  probit  presented  in  Chapter  IV.  The  same  variables 
are  significant  to  the  one  percent  level.  The  only  difference  is  the  sign  for  the  measure  of 
industry  concentration  has  flipped  from  positive  to  negative.  However,  the  statistical 
significance  on  both  models  is  low  enough  to  make  this  result  insignificant.  Therefore, 
the  model  is  not  sensitive  to  the  specific  measure  of  industry  concentration. 
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